Mobile robot and control method for controlling the same

ABSTRACT

A mobile robot includes a travel drive unit configured to move a main body, an image acquisition unit configured to acquire an image of surroundings of the main body, a storage configured to store the image acquired by the image acquisition unit, a sensor unit having one or more sensors configured to sense an object during the movement of the body, and a controller configured to perform control to extract a part of the image acquired by the image acquisition unit in correspondence to a direction in which the object sensed by the sensor unit is present.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a U.S. National Phase entry under 35 U.S.C. § 371from PCT International Application No. PCT/KR2017/009257, filed Aug. 24,2017, which claims the benefit of priority of Korean Patent ApplicationsNos. 10-2016-0108384, 10-2016-0108385, and 10-2016-0108386, all filedAug. 25, 2016, the contents of which are incorporated herein byreference in their entireties.

TECHNICAL FIELD

The present invention relates to a mobile robot and a method forcontrolling the same, and, more particularly, to a mobile robot capableof performing object recognition and avoidance and a method forcontrolling the mobile robot.

BACKGROUND ART

Robots have been developed for industry and has taken charge of aportion of factory automation. As the robots have recently been used inmore diverse fields, medical robots, aerospace robots, etc. have beendeveloped, and domestic robots for ordinary households are also beingmade. Among those robots, a robot capable of traveling on its own isreferred to as a mobile robot.

A typical example of mobile robots used in households is a robotcleaner, which cleans a predetermined area by suctioning ambient dust orforeign substances while traveling in the predetermined area on its own.

A mobile robot is capable of moving on its own and thus has freedom tomove, and the mobile robot is provided with a plurality of sensors toavoid an object during traveling so that the mobile robot may travelwhile detouring around the object.

In general, an infrared sensor or an ultrasonic sensor is used to sensean object by the mobile robot. The infrared sensor determines presenceof an object and a distance to the object based on an amount of lightreflected by the object or a time taken to receive the reflected light,whereas the ultrasonic sensor emits an ultrasound wave having apredetermined period and, in response to ultrasonic waves reflected byan object, determines a distance to the object based on a differencebetween the time to emit the ultrasonic waves and the time for theultrasonic waves to return to the object.

Meanwhile, object recognition and avoidance significantly affects notjust driving performance but also cleaning performance of the mobilerobot, and thus, it is required to secure reliability of an objectrecognizing capability.

A related art (Korean Patent Registration No. 10-0669892) discloses atechnology that embodies highly reliable object recognition by combiningan infrared sensor and an ultrasonic sensor.

However, the related art (Korean Patent Registration No. 10-0669892) hasa problem in its incapability of determining an attribute of an object.

FIG. 1 is a diagram for explanation of a method in which a conventionalmobile robot senses and avoids an object.

Referring to FIG. 1, a robot cleaner performs cleaning by suctioningdust and foreign substance while moving (S11).

The robot cleaner recognizes presence of an object in response tosensing, by an ultrasonic sensor, an ultrasonic signal reflected by anobject (S12), and determines whether the recognized object has a heightthat the robot cleaner is able to climb (S13).

If it is determined that the recognized object has a height that therobot cleaner is able to climb, the robot cleaner may move in a straightforward direction (S14), and, otherwise, rotate by an angle of 90degrees (S15).

For example, if the object is a low door threshold, the robot cleanerrecognizes the door threshold, and, if a recognition result indicatesthat the robot cleaner is able to pass the door threshold, the robotcleaner moves over the door threshold.

However, if an object determined to have a height that the mobile robotis able to climb is an electric wire, the robot cleaner may be confinedthus restricted by the electric wire when moving over the electric wire.

In addition, as a fan base has a height similar to or lower than aheight of a door threshold, the robot cleaner may determine that the fanbase is an object that the robot cleaner is able to climb. In this case,when attempting to climb the fan base, the robot cleaner may berestricted with wheels spinning.

In addition, in the case where, not an entire human body, but a part ofhuman hair is sensed, the robot cleaner may determine that the humanhair has a height that the robot cleaner is able to climb, and travelforward, but, in this case, the robot cleaner may suction the human hairwhich possibly cause an accident.

Thus, a method for identifying an attribute of a front object and changea moving pattern according to the attribute is required.

Meanwhile, interests for machine learning such as artificialintelligence and deep learning have grown significantly in recent years.

Machine learning is one of artificial intelligence fields, and refers totraining computer with data and instruct the computer to perform acertain task, such as prediction and classification, based on the data.

The representative definition of machine learning is proposed byProfessor Tom M. Mitchell, as follows: “if performance at a specifictask in T, as measured by P, improves with experience E, it can be saidthat the task T is learned from experience E.” That is, improvingperformance with respect to a certain task T with consistent experienceE is considered machine learning.

Conventional machine learning focuses on statistics-basedclassification, regression, and clustering models. In particular,regarding map learning of the classification and regression models, ahuman defines properties of training data and a learning model ofdistinguishing new data based on the properties.

Meanwhile, unlike the conventional machine learning, deep learningtechnologies have been being developed recently. Deep learning is anartificial intelligence technology that enables a computer to learn onits own like a human on the basis of Artificial Neural Networks (ANN).That is, deep learning means that a computer discovers and determinesproperties on its own.

One of factors having accelerated development of deep learning may be,for example, deep learning framework that is provided as an open source.For example, deep learning frameworks include Theano developed byUniversity of Montreal, Canada, Torch developed by New York University,U.S.A., Caffe developed by University of California, Berkeley,TensorFlow developed by Google, etc.

With disclosure of deep learning frameworks, a learning process, alearning method, and extracting and selecting data used for learninghave become more important in addition to a deep learning algorithm forthe purposes of effective learning and recognition.

In addition, more and more efforts are being made to use artificialintelligence and machine learning in a diversity of products andservices.

DISCLOSURE Technical Problem

One object of the present invention is to provide a mobile robot and amethod for controlling the mobile robot, the mobile robot which iscapable of acquiring image data for improving accuracy in objectattribute recognition.

Another object of the present invention is to provide a mobile robot anda method for controlling the mobile robot, the mobile robot which iscapable of determining an attribute of an object and adjusting atraveling pattern according to the attribute of the object to therebyrecognize and avoid the object with high reliability.

Yet another object of the present invention is to provide a mobilerobot, which is capable of moving forward, moving backward, stopping,detouring, and the like according to a recognition result on an objectto enhance stability of the mobile robot and user convenience andimprove operational efficiency and cleaning efficiency, and a method forcontrolling the mobile robot.

Yet another object of the present invention is to provide a mobile robotcapable of accurately recognizing an attribute of an object based onmachine learning, and a method for controlling the mobile robot.

Yet another object of the present invention is to provide a mobile robotcapable of performing machine learning efficiently and extracting datafor object attribute recognition, and a method for controlling themobile robot.

Technical Solution

In order to achieve the above or other objects, a mobile robot in onegeneral aspect of the present invention includes: a travel drive unitconfigured to move a main body; an image acquisition unit configured toacquire an image of surroundings of the main body; a storage configuredto store the image acquired by the image acquisition unit; a sensor unithaving one or more sensors configured to sense an object during themovement of the main body; and a controller configured to performcontrol to extract a part of the image acquired by the image acquisitionunit in correspondence to a direction in which the object sensed by thesensor unit is present, and accordingly, it is possible to extract datathat is efficient for machine learning and object attribute recognition.

In addition, in order to achieve the above or other objects, a mobilerobot in one general aspect of the present invention includes an objectrecognition module configured to recognize, in an extracted image, anobject pre-learned through machine learning and a travel control moduleconfigured to control driving of a travel drive unit based on anattribute of the recognized object, and accordingly, it is possible toimprove stability, user convenience, operation efficiency, and cleaningefficiency.

In addition, in order to achieve the above or other objects, a methodfor controlling a mobile robot in one general aspect of the presentinvention may include: sensing, by a sensor unit, object duringmovement; extracting a part of an image acquired by an image acquisitionunit in correspondence to a direction in which the object sensed by thesensor unit is present; recognizing an object in the extracted part ofthe image; and controlling driving of a travel drive unit based on anattribute of the recognized object.

Advantageous Effects

According to at least one of the embodiments of the present invention,it is possible to acquire image data that helps increase accuracy ofrecognizing an attribute of an object.

According to at least one of the embodiments of the present invention,since a mobile robot is able to to determine an attribute of an objectand adjust a traveling pattern according to the attribute of the object,it is possible to perform highly reliable object recognition andavoidance operation.

In addition, according to at least one of the embodiments of the presentinvention, it is possible to provide a mobile robot which is capable ofmoving forward, moving backward, stopping, detouring, and the likeaccording to a recognition result on an object to enhance userconvenience and improve operational efficiency and cleaning efficiency,and a method for controlling the mobile robot.

In addition, according to at least one of the embodiments of the presentinvention, it is possible to provide a mobile robot and a method forcontrolling the mobile robot, the mobile robot which is capable ofaccurately recognizing an attribute of an object based on learningmachine.

In addition, according to at least one of the embodiments of the presentinvention, it is possible for a mobile robot to perform machine learningefficiently and extract data for recognition.

Meanwhile, other effects may be explicitly or implicitly disclosed inthe following detailed description according to embodiments of thepresent invention.

DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram for explanation of a method for sensing and avoidingobject by a conventional robot cleaner.

FIG. 2 is a perspective view of a mobile robot according to anembodiment of the present invention and a base station for charging themobile robot.

FIG. 3 is a view of the top section of the mobile robot in FIG. 2.

FIG. 4 is a view of the front section of the mobile robot in FIG. 2.

FIG. 5 is a view of the bottom section of the mobile robot in FIG. 2.

FIGS. 6 and 7 are block diagram of a control relationship between majorcomponents of a mobile robot according to an embodiment of the presentinvention.

FIG. 8 is an exemplary schematic internal block diagram of a serveraccording to an embodiment of the present invention.

FIGS. 9 to 12 are diagrams for explanation of deep learning.

FIG. 13 is a flowchart of a method for controlling a mobile robotaccording to an embodiment of the present invention.

FIG. 14 is a flowchart of a method for controlling a mobile robotaccording to an embodiment of the present invention.

FIGS. 15 to 21 are diagrams for explanation of a method for controllinga mobile robot according to an embodiment of the present invention.

FIG. 22 is a diagram for explanation of an operation method of a mobilerobot and a server according to an embodiment of the present invention.

FIG. 23 is a flowchart of an operation method of a mobile robot and aserver according to an embodiment of the present invention.

FIGS. 24 to 26 are diagrams for explanation of a method for controllinga moving robot according to an embodiment of the present invention.

FIG. 27 is a flowchart of a method for controlling a mobile robotaccording to an embodiment of the present invention.

FIG. 28 is a diagram for explanation of a method for controlling amobile robot according to an embodiment of the present invention.

FIG. 29 is a flowchart of a method for controlling a mobile robotaccording to an embodiment of the present invention.

FIGS. 30 to 34 are diagrams for explanation of a method for controllinga mobile robot according to an embodiment of the present invention

BEST MODE

Hereinafter, embodiments of the present invention will be described indetail with reference to the accompanying drawings. While the inventionwill be described in conjunction with exemplary embodiments, it will beunderstood that the present description is not intended to limit theinvention to the exemplary embodiments.

In the drawings, in order to clearly and briefly describe the invention,parts which are not related to the description will be omitted, and, inthe following description of the embodiments, the same or similarelements are denoted by the same reference numerals even though they aredepicted in different drawings.

In the following description, with respect to constituent elements usedin the following description, the suffixes “module” and “unit” are usedor combined with each other only in consideration of ease in thepreparation of the specification, and do not have or serve as differentmeanings. Accordingly, the suffixes “module” and “unit” may beinterchanged with each other.

A mobile robot 100 according to an embodiment of the present inventionrefers to a robot capable of moving on its own using wheels and thelike, and may be a domestic robot, a robot cleaner, etc. A robot cleanerhaving a cleaning function is herein after described as an example ofthe mobile robot, but the present invention is not limited thereto.

FIG. 2 is a perspective view of a mobile robot and a charging stationfor charging the mobile robot according to an embodiment of the presentinvention.

FIG. 3 is a view of the top section of the mobile robot illustrated inFIG. 2, FIG. 4 is a view of the front section of the mobile robotillustrated in FIG. 2, and FIG. 5 is a view of the bottom section of themobile robot illustrated in FIG. 2.

FIGS. 6 and 7 are block diagram of a control relationship between majorcomponents of a mobile robot according to an embodiment of the presentinvention.

Referring to FIGS. 2 to 7, a mobile robot 100, 100 a, or 100 b includesa main body 110, and an image acquisition unit 120, 120 a, and 120 b foracquiring an area around the main body 110.

Hereinafter, each part of the main body 110 is defined as below: aportion facing a ceiling of a cleaning area is defined as the topsection (see FIG. 3), a portion facing a floor of the cleaning area isdefined as the bottom section (see FIG. 5), and a portion facing adirection of travel in portions constituting a circumference of the mainbody 110 between the top section and the bottom section is defined asthe front section (see FIG. 4).

The mobile robot 100, 100 a, or 100 b includes a travel drive unit 160for moving the main body 110. The travel drive unit 160 includes atleast one drive wheel 136 for moving the main body 110. The travel driveunit 160 includes a driving motor (not shown) connected to the drivewheel 136 to rotate the drive wheel. Drive wheels 136 may be provided onthe left side and the right side of the main body 110, respectively, andsuch drive wheels 136 are hereinafter referred to as a left wheel 136(L)and a right wheel 136(R), respectively.

The left wheel 136(L) and the right wheel 136(R) may be driven by onedriving motor, but, if necessary, a left wheel drive motor to drive theleft wheel 136(L) and a right wheel drive motor to drive the right wheel136(R) may be provided. The travel direction of the main body 110 may bechanged to the left or to the right by making the left wheel 136(L) andthe right wheel 136(R) have different rates of rotation.

A suction port 110 h to suction air may be formed on the bottom surfaceof the body 110, and the body 110 may be provided with a suction device(not shown) to provide suction force to cause air to be suctionedthrough the suction port 110 h, and a dust container (not shown) tocollect dust suctioned together with air through the suction port 110 h.

The body 110 may include a case 111 defining a space to accommodatevarious components constituting the mobile robot 100, 110 a, 110 b. Anopening allowing insertion and retrieval of the dust containertherethrough may be formed on the case 111, and a dust container cover112 to open and close the opening may be provided rotatably to the case111.

There may be provided a roll-type main brush having bristles exposedthrough the suction port 110 h and an auxiliary brush 135 positioned inthe front of the bottom surface of the body 110 and having bristlesforming a plurality of radially extending blades. Dust is removed fromthe floor in a cleaning area by rotation of the brushes 134 and 135, andsuch dust separated from the floor in this way is suctioned through thesuction port 110 h and collected in the dust container.

A battery 138 serves to supply power necessary not only for the drivemotor but also for overall operations of the mobile robot 100, 100 a, or100 b. When the battery 138 of the robot cleaner 100 is running out, themobile robot 100, 100 a, or 100 b may perform return travel to thecharging base 200 to charge the battery, and during the return travel,the robot cleaner 100 may autonomously detect the position of thecharging base 200.

The charging base 200 may include a signal transmitting unit (not shown)to transmit a predetermined return signal. The return signal mayinclude, but is not limited to, a ultrasonic signal or an infraredsignal.

The mobile robot 100, 100 a, or 100 b may include a signal sensing unit(not shown) to receive the return signal. The charging base 200 maytransmit an infrared signal through the signal transmitting unit, andthe signal sensing unit may include an infrared sensor to sense theinfrared signal. The mobile robot 100, 100 a, or 100 b moves to theposition of the charging base 200 according to the infrared signaltransmitted from the charging base 200 and docks with the charging base200. By docking, charging of the robot cleaner 100 is performed betweena charging terminal 133 of the mobile robot 100, 100 a, or 100 b and acharging terminal 210 of the charging base 200.

In some implementations, the mobile robot 100, 100 a, or 100 b mayperform return travel to the base station 200 based on an image or in alaser-pattern extraction method.

The mobile robot 100, 100 a, or 100 b may return to the base station byrecognizing a predetermined pattern formed at the base station 200 usingan optical signal emitted from the main body 110 and extracting thepredetermined pattern.

For example, the mobile robot 100, 100 a, or 100 b according to anembodiment of the present invention may include an optical patternsensor (not shown).

The optical pattern sensor may be provided in the main body 110, emit anoptical pattern to an active area where the mobile robot 100, 100 a, or100 b moves, and acquire an input image by photographing an area intowhich the optical pattern is emitted. For example, the optical patternmay be light having a predetermined pattern, such as a cross pattern.

The optical pattern sensor may include a pattern emission unit to emitthe optical pattern, and a pattern image acquisition unit to photographan area into which the optical pattern is emitted.

The pattern emission unit may include a light source, and an OpticalPattern Projection Element (OPPE). The optical pattern is generated aslight incident from the light source passes through the OPPE. The lightsource may be a Laser Diode (LD), a Light Emitting Diode (LED), etc.

The pattern emission unit may emit light in a direction forward of themain body, and the pattern image acquisition unit acquires an inputimage by photographing an area where an optical pattern is emitted. Thepattern image acquisition unit may include a camera, and the camera maybe a structured light camera.

Meanwhile, the base station 200 may include two or more locator beaconsspaced a predetermined distance apart from each other.

A locator beacon forms a mark distinguishable from the surroundings whenan optical pattern is incident on a surface of the locator beacon. Thismark may be caused by transformation of the optical pattern due to amorphological characteristic of the locator beacon, or may be caused bydifference in light reflectivity (or absorptivity) due to a materialcharacteristic of the locator beacon.

The locator beacon may include an edge that forms the mark. An opticalpattern incident on the surface of the locator beacon is bent on theedge with an angle, and the bending is found as a cusp in an input imageas the mark.

The mobile robot 100, 100 a, or 100 b may automatically search for thebase station when a remaining battery capacity is insufficient, or maysearch for the base station even when a charging command is receivedfrom a user.

When the mobile robot 100, 100 a, or 100 b searches for the basestation, a pattern extraction unit extracts cusps from an input imageand a controller 140 acquire location information of the extractedcusps. The location information may include locations in athree-dimensional space, which takes into consideration of distancesfrom the mobile robot 100, 100 a, or 100 b to the cusps.

The controller 140 calculates an actual distance between the cusps basedon the acquired location information of the cusps and compare the actualdistance with a preset reference threshold, and, if a distance betweenthe actual distance and the reference threshold falls into apredetermined range, it may be determined that the charging station 200is found.

Alternatively, the mobile robot 100, 100 a, or 100 b may return to thebase station 200 by acquiring images of the surroundings with a cameraof the image acquisition unit 120 and extracting and identifying a shapecorresponding to the charging station 200 from the acquired image.

In addition, the mobile robot 100, 100 a, or 100 b may return to thebase station by acquiring images of surroundings with the camera of theimage acquisition unit 120 and identifying a predetermined opticalsignal emitted from the base station 200.

The image acquisition unit 120, which is configured to photograph thecleaning area, may include a digital camera. The digital camera mayinclude at least one optical lens, an image sensor (e.g., a CMOS imagesensor) including a plurality of photodiodes (e.g., pixels) on which animage is created by light transmitted through the optical lens, and adigital signal processor (DSP) to construct an image based on signalsoutput from the photodiodes. The DSP may produce not only a still image,but also a video consisting of frames constituting still images.

Preferably, the image acquisition unit 120 includes a front camera 120 aprovided to acquire an image of a front field of view from the main body110, and an upper camera 120 b provided on the top section of the mainbody 110 to acquire an image of a ceiling in a travel area, but theposition and capture range of the image acquisition unit 120 are notnecessarily limited thereto.

In this embodiment, a camera may be installed at a portion (ex. thefront, the rear, and the bottom) of the mobile robot and continuouslyacquire captured images in a cleaning operation. Such a camera may beinstalled in plural at each portion for photography efficiency. An imagecaptured by the camera may be used to recognize a type of a substanceexisting in a corresponding space, such as dust, a human hair, andfloor, to determine whether cleaning has been performed, or to confirm apoint in time to clean.

The front camera 120 a may capture an object or a condition of acleaning area ahead in a direction of travel of the mobile robot 100,100 a, or 100 b.

According to an embodiment of the present invention, the imageacquisition unit 120 may acquire a plurality of images by continuouslyphotographing the surroundings of the main body 110, and the pluralityof acquired images may be stored in a storage 150.

The mobile robot 100, 100 a, or 100 b may increase accuracy in objectrecognition by using a plurality of images or may increase accuracy ofrecognition of an object by selecting one or more images from aplurality of images to use effective data.

In addition, the mobile robot 100, 100 a, or 100 b may include a sensorunit 170 that includes sensors serving to sense a variety of datarelated to operation and states of the mobile robot.

For example, the sensor unit 170 may include an object detection sensorto sense an object present ahead. In addition, the sensor unit 170 mayfurther include a cliff detection sensor 132 to sense presence of acliff on the floor of a travel region, and a lower camera sensor 139 toacquire an image of the floor.

Referring to FIGS. 2 and 4, the object detection sensor 131 may includea plurality of sensors installed at a predetermined interval on an outercircumferential surface of the mobile robot 100.

For example, the sensor unit 170 may include a first sensor disposed onthe front surface of the main body 110, and second and third sensorsspaced apart from the first sensor in the left and right direction.

The object detection sensor 131 may include an infrared sensor, anultrasonic sensor, an RF sensor, a magnetic sensor, a position sensitivedevice (PSD) sensor, etc.

Meanwhile, the positions and types of sensors included in the objectdetection sensor 131 may differ depending on a model of a mobile robot,and the object detection sensor 131 may further include various sensors.

The object detection sensor 131 is a sensor for sensing a distance to awall or an object in an indoor space, and is, but not limited thereto,described as an ultrasonic sensor.

The object detection sensor 131 senses an object, especially an object,present in a direction of travel (movement) of the mobile robot andtransmits object information to the controller 140. That is, the objectdetection sensor 131 may sense a protrusion, home stuff, furniture, awall surface, a wall corner, etc. that are present in a path throughwhich the robot cleaner moves, and deliver related information to acontrol unit.

In this case, the controller 140 may sense a position of the objectbased on at least two signals received through the ultrasonic sensor,and control movement of the mobile robot 100 according to the sensedposition of the object.

In some implementations, the object detection sensor 131 provided on anouter side surface of the case 110 may include a transmitter and areceiver.

For example, the ultrasonic sensor may be provided such that at leastone transmitter and at least two receivers cross one another.Accordingly, it is possible to transmit a signal in various directionsand receive a signal, reflected by an object, from various directions.

In some implementations, a signal received from the object detectionsensor 131 may go through signal processing, such as amplification andfiltering, and thereafter a distance and a direction to an object may becalculated.

Meanwhile, the sensor unit 170 may further include an operationdetection sensor that serves to sense operation of the mobile robot 100,100 a, or 100 b upon driving of the main body 110, and output operationinformation. As the operation detection sensor, a gyro sensor, a wheelsensor, an acceleration sensor, etc. may be used.

When the mobile robot 100, 100 a, or 100 b moves according to anoperation mode, the gyro sensor senses a direction of rotation anddetects an angle of rotation. The mobile robot 100, 100 a, or 100 bdetects a velocity of the mobile robot 100, 100 a, or 100 b, and outputsa voltage value proportional to the velocity. The controller 140 maycalculate the direction of rotation and the angle of rotation using thevoltage value that is output from the gyro sensor.

The wheel sensor is connected to the left wheel 136(L) and the rightwheel 136(R) to sense the revolution per minute (RPM) of the wheels. Inthis example, the wheel sensor may be a rotary encoder. The rotaryencoder senses and outputs the RPM of the left wheel 136(L) and theright wheel 136(R).

The controller 140 may calculate the speed of rotation of the left andright wheels using the RPM. In addition, the controller 140 maycalculating an angle of rotation using a difference in the RPM betweenthe left wheel 136(L) and the right wheel 136(R).

The acceleration sensor senses a change in speed of the mobile robot100, 100 a, or 100 b, e.g., a change in speed according to a start, astop, a direction change, collision with an object, etc. Theacceleration sensor may be attached to a position adjacent to a mainwheel or a secondary wheel to detect the slipping or idling of thewheel.

In addition, the acceleration sensor may be embedded in the controller140 and detect a change in speed of the mobile robot 100, 100 a, or 100b. That is, the acceleration sensor detects impulse according to achange in speed and outputs a voltage value corresponding thereto. Thus,the acceleration sensor may perform the function of an electronicbumper.

The controller 140 may calculate a change in the positions of the mobilerobot 100, 100 a, or 100 b based on operation information output fromthe operation detection sensor. Such a position is a relative positioncorresponding to an absolute position that is based on imageinformation. By recognizing such a relative position, the mobile robotmay enhance position recognition performance using image information andobject information.

Meanwhile, the mobile robot 100, 100 a, or 100 b may include a powersupply (not shown) having a rechargeable battery 138 to provide power tothe robot cleaner.

The power supply may supply driving power and operation power to eachelement of the mobile robot 100, 100 a, or 100 b, and may be charged byreceiving a charging current from the charging station 200 when aremaining power capacity is insufficient.

The mobile robot 100, 100 a, or 100 b may further include a batterysensing unit to sense the state of charge of the battery 138 andtransmit a sensing result to the controller 140. The battery 138 isconnected to the battery sensing unit such that a remaining batterycapacity state and the state of charge of the battery is transmitted tothe controller 140. The remaining battery capacity may be disposed on ascreen of an output unit (not shown).

In addition, the mobile robot 100, 100 a, or 100 b includes an operator137 to input an on/off command or any other various commands. Throughthe operator 137, various control commands necessary for overalloperations of the mobile robot 100 may be received. In addition, themobile robot 100, 100 a, or 100 b may include an output unit (not shown)to display reservation information, a battery state, an operation mode,an operation state, an error state, etc.

Referring to FIGS. 6 and 7, the mobile robot 100 a, or 100 b includesthe controller 140 for processing and determining a variety ofinformation, such as recognizing the current position, and the storage150 for storing a variety of data. In addition, the mobile robot 100,100 a, or 100 b may further include a communication unit 190 fortransmitting and receiving data with an external terminal

The external terminal may be provided with an application forcontrolling the mobile robot 100 a and 100 b, displays a map of a travelarea to be cleaned by executing the application, and designate aspecific area on a map to be cleaned. Examples of the external terminalmay include a remote control, a PDA, a laptop, a smartphone, a tablet,and the like in which an application for configuring a map is installed.

By communicating with the mobile robot 100 a and 100 b, the externalterminal may display the current position of the mobile robot, inaddition to the map, and display information on a plurality of areas. Inaddition, the external terminal may update the current position inaccordance with traveling of the mobile robot and displays the updatedcurrent position.

The controller 140 controls the image acquisition unit 120, the operator137, and the travel drive unit 160 of the mobile robot 100 a and 100 bso as to control overall operations of the mobile robot 100.

The storage 150 serves to record various kinds of information necessaryfor control of the robot cleaner 100 and may include a volatile ornon-volatile recording medium. The recording medium serves to store datawhich is readable by a micro processor and may include a hard disk drive(HDD), a solid state drive (SSD), a silicon disk drive (SDD), a ROM, aRAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage,etc.

Meanwhile, a map of the travel area may be stored in the storage 150.The map may be input by an external terminal or a server capable ofexchanging information with the mobile robot 100 a and 100 b throughwired or wireless communication, or may be constructed by the mobilerobot 100 a and 100 b through self-learning.

On the map, positions of rooms within the cleaning area may be marked.In addition, the current position of the mobile robot 100 a and 100 bmay be marked on the map, and the current position of the mobile robot100 a and 100 b on the map may be updated during travel of the mobilerobot 100 a and 100 b. The external terminal stores a map identical to amap stored in the storage 150.

The storage 150 may store cleaning history information. The cleaninghistory information may be generated each time cleaning is performed.

The map of a travel area stored in the storage 150 may be a navigationmap used for travel during cleaning, a simultaneous localization andmapping (SLAM) map used for position recognition, a learning map usedfor learning cleaning by storing corresponding information uponcollision with an object or the like, a global localization map used forglobal localization recognition, an object recognition map whereinformation on a recognized object is recorded, etc.

Meanwhile, maps may be stored and managed in the storage 150 by purposesor may be classified exactly by purposes. For example, a plurality ofinformation items may be stored in one map so as to be used for at leasttwo purposes. For an instance, information on a recognized object may berecorded on a learning map so that the learning map may replace anobject recognition map, and a global localization position is recognizedusing the SLAM map used for position recognition so that the SLAM mapmay replace or be used together with the global localization map.

The controller 140 may include a travel control module 141, a positionrecognition module 142, a map generation module 143, and an objectrecognition module 144.

Referring to FIGS. 3 to 7, the travel control module 141 serves tocontrol travelling of the mobile robot 100, 100 a, or 100 b, andcontrols driving of the travel drive unit 160 according to a travelsetting. In addition, the travel control module 141 may determine atravel path of the mobile robot 100, 100 a, or 100 b based on operationof the travel drive unit 160. For example, the travel control module 141may determine the current or previous moving speed or distance travelledof the mobile robot 100 based on an RPM of the drive wheel 136, and mayalso determine the current or previous direction shifting process basedon directions of rotation of each drive wheel 136(L) or 136(R). Based ontravel information of the mobile robot 100, 100 a, or 100 b determinedin this manner, the position of the mobile robot 100, 100 a, or 100 b onthe map may be updated.

The map generation module 143 may generate a map of a travel area. Themap generation module 143 may write a map by processing an imageacquired by the image acquisition unit 120. That is, a cleaning mapcorresponding to a cleaning area may be written.

In addition, the map generation module 143 may enable globallocalization in association with a map by processing an image acquiredby the image acquisition unit 120 at each position.

The position recognition module 142 may estimate and recognize thecurrent position. The position recognition module 142 identifies theposition in association with the map generation module 143 using imageinformation of the image acquisition unit 120, thereby enabled toestimate and recognize the current position even when the position ofthe mobile robot 100, 100 a, or 100 b suddenly changes.

The mobile robot 100, 100 a, or 100 b may recognize the position usingthe position recognition module 142 during continuous travel, and theposition recognition module 142 may learn a map and estimate the currentposition by using the map generation module 143 and the objectrecognition module 144 without the position recognition module 142.

While the mobile robot 100, 100 a, or 100 b travels, the imageacquisition unit 120 acquires images of the surroundings of the mobilerobot 100. Hereinafter, an image acquired by the image acquisition unit120 is defined as an “acquisition image”.

An acquisition image includes various features located at a ceiling,such as lighting devices, edges, corners, blobs, ridges, etc.

The map generation module 143 detects features from each acquisitionimage. In computer vision, there are a variety of well-known techniquesfor detecting features from an image. A variety of feature detectorssuitable for such feature detection are well known. For example, thereare Canny, Sobel, Harris&Stephens/Plessey, SUSAN, Shi&Tomasi, Levelcurve curvature, FAST, Laplacian of Gaussian, Difference of Gaussians,Determinant of Hessian, MSER, PCBR, and Grey-level blobs detectors.

The map generation module 143 calculates a descriptor based on eachfeature. For feature detection, the map generation module 143 mayconvert a feature into a descriptor using Scale Invariant FeatureTransform (SIFT).

Meanwhile, the descriptor is defined as a group of individual featurespresent in a specific space and may be expressed as an n-dimensionalvector. For example, various features at a ceiling, such as edges,corners, blobs, ridges, etc. may be calculated into respectivedescriptors and stored in the storage 150.

Based on descriptor information acquired from an image of each position,at least one descriptor is classified by a predetermined subordinateclassification rule into a plurality of groups, and descriptors includedin the same group may be converted by a predetermined subordinaterepresentation rule into subordinate representative descriptors. Thatis, a standardization process may be performed by designating arepresentative value for descriptors obtained from an individual image.

The SIFT enables detecting a feature invariant to a scale, rotation,change in brightness of a subject, and thus, it is possible to detect aninvariant (i.e., rotation-invariant) feature of an area even when imagesof the area is captured by changing a position of the mobile robot 100.However, aspects of the present invention are not limited thereto, andVarious other techniques (e.g., Histogram of Oriented Gradient (HOG),Haar feature, Fems, Local Binary Pattern (LBP), and Modified CensusTransform (MCT)) can be applied.

The map generation module 143 may classify at least one descriptor ofeach acquisition image into a plurality groups based on descriptorinformation, obtained from an acquisition image of each position, by apredetermined subordinate classification rule, and may convertdescriptors included in the same group into subordinate representativedescriptors by a predetermined subordinate representation rule.

In another example, all descriptors obtained from acquisition images ofa predetermined area, such as a room, may be classified into a pluralityof groups by a predetermined subordinate classification rule, anddescriptors included in the same group may be converted into subordinaterepresentative descriptors by the predetermined subordinaterepresentation rule.

Through the above procedure, the map generation module 143 may obtain afeature distribution for each position. The feature distribution of eachposition may be represented by a histogram or an n-dimensional vector.In another example, without using the predetermined subordinateclassification rule and the predetermined subordinate representationrule, the map generation module 143 may estimate an unknown currentposition based on a descriptor calculated from each feature.

In addition, when the current position of the mobile robot 100, 100 a,or 100 b has become unknown due to position hopping, the map generationmodule may estimate the current position of the mobile robot 100 basedon data such as a pre-stored descriptor and a subordinate representativedescriptor.

The mobile robot 100, 100 a, or 100 b acquires an acquisition unit imageat the unknown current position using the image acquisition unit 120.Various features located at a ceiling, such as lighting devices, edges,corners, blobs, and ridges, are found in the acquisition image.

The position recognition module 142 detects features from an acquisitionimage. Descriptions about various methods for detecting features from animage in computer vision and various feature detectors suitable forfeature detection are the same as described above.

The position recognition module 142 calculates a recognition descriptorbased on each recognition feature through a recognition descriptorcalculating process. In this case, the recognition feature and therecognition descriptor are used to describe a process performed by theobject recognition module 144, and to be distinguished from the termsdescribing the process performed by the map generation module 143.However, these are merely terms defined to describe characteristics of aworld outside the mobile robot 100, 100 a, or 100 b.

For such feature detection, the position recognition module 142 mayconvert a recognition feature into a recognition descriptor using theSIFT technique. The recognition descriptor may be represented by ann-dimensional vector.

The SIFT is an image recognition technique by which easilydistinguishable features, such as corners, are selected in anacquisition image and n-dimensional vectors are obtained, which arevalues of dimensions indicative of a degree of radicalness of change ineach direction with respect to a distribution feature (a direction ofbrightness change and a degree of radicalness of the change) of gradientof pixels included in a predetermined area around each of the features.

Based on information on at least one recognition descriptor obtainedfrom an acquisition image acquired at an unknown current position, theposition recognition module 142 performs conversion by a predeterminedsubordinate conversion rule into information (a subordinate recognitionfeature distribution), which is comparable with position information(e.g., a subordinate feature distribution for each position) subject tocomparison.

By the predetermined subordinate comparison rule, a feature distributionfor each position may be compared with a corresponding recognitionfeature distribution to calculate a similarity therebetween. Asimilarity (probability) for the position corresponding to eachposition, and a position having the highest probability may bedetermined to be the current position.

As such, the controller 140 may generate a map of a travel areaconsisting of a plurality of areas, or recognize the current position ofthe main body 110 based on a pre-stored map.

When the map is generated, the controller 140 transmits the map to anexternal terminal through the communication unit 190. In addition, asdescribed above, when a map is received from the external terminal, thecontroller 140 may store the map in the storage.

In addition, when the map is updated during traveling, the controller140 may transmit update information to the external terminal so that themap stored in the external terminal becomes identical to the map storedin the mobile robot 100. As the identical map is stored in the externalterminal and the mobile robot 100, the mobile robot 100 may clean adesignated area in response to a cleaning command from a mobileterminal, and the current position of the mobile robot may be displayedin the external terminal.

In this case, the map divides a cleaning area into a plurality of areas,includes a connection channel connecting the plurality of areas, andincludes information on an object in an area.

When a cleaning command is received, the controller 140 determineswhether a position on the map and the current position of the mobilerobot match each other. The cleaning command may be received from aremote controller, an operator, or the external terminal.

When the current position does not match with a position marked on themap, or when it is not possible to confirm the current position, thecontroller 140 recognizes the current position, recovers the currentposition of the mobile robot 100 and then controls the travel drive unit160 to move to a designated area based on the current position.

When the current position does not match with a position marked on themap or when it is not possible to confirm the current position, theposition recognition module 142 analyzes an acquisition image receivedfrom the image acquisition unit 120 and estimate the current positionbased on the map. In addition, the object recognition module 144 or themap generation module 143 may recognize the current position asdescribed above.

After the current position of the mobile robot 100, 100 a, or 100 b isrecovered by recognizing the position, the travel control module 141 maycalculate a travel path from the current position to a designated areaand controls the travel drive unit 160 to move to the designated area.

When a cleaning pattern information is received from a serer, the travelcontrol module 141 may divide the entire travel area into a plurality ofareas based on the received cleaning pattern information and set one ormore areas as a designated area.

In addition, the travel control module 141 may calculate a travel pathbased on the received cleaning pattern information, and perform cleaningwhile traveling along the travel path.

When cleaning of a set designated area is completed, the controller 140may store a cleaning record in the storage 150.

In addition, through the communication unit 190, the controller 140 maytransmit an operation state or a cleaning state of the mobile robot 100at a predetermined cycle.

Accordingly, based on the received data, the external terminal displaysthe position of the mobile robot together with a map on a screen of anapplication in execution, and outputs information on a cleaning state.

The mobile robot 100, 100 a, or 100 b moves in one direction untildetecting an object or a wall, and, when the object recognition module144 recognizes an object, the mobile robot 100, 100 a, or 100 b maydetermine a travel pattern, such as forward travel and rotation,depending on an attribute of the recognized object.

For example, if the attribute of the recognized object implies an objectthat the mobile robot is able to climb, the mobile robot 100, 100 a, or100 b may keep traveling straight forward. Alternatively, if theattribute of the recognized object implies an object that the mobilerobot is not able to climb, the mobile robot 100, 100 a, or 100 b maytravel in a zigzag pattern in a manner in which the mobile robot 100,100 a, or 100 b rotates, moves a predetermined distance, and then movesa distance in a direction opposite to the initial direction of traveluntil detecting another object.

The mobile robot 100, 100 a, or 100 b may perform object recognition andavoidance based on machine learning.

The controller 140 may include the object recognition module 144 forrecognizing an object, which is pre-learned through machine learning, inan input image, and the travel control module 141 for controllingdriving of the travel drive unit 160 based on an attribute of therecognized object.

The mobile robot 100, 100 a, or 100 b may include the object recognitionmodule 144 that has learned an attribute of an object through machinelearning.

The machine learning refers to a technique which enables a computer tolearn without a logic instructed by a user so as to solves a problem onits own.

Deep learning is an artificial intelligence technology that trains acomputer to learn human thinking based on Artificial Neural Networks(ANN) for constructing artificial intelligence so that the computer iscapable of learning on its own without a user's instruction.

The ANN may be implemented in a software form or a hardware form, suchas a chip and the like.

The object recognition module 144 may include an ANN in a software formor a hardware form, in which attributes of objects is learned.

For example, the object recognition module 144 may include a Depp NeuralNetwork (DNN) trained through deep learning, such as ConvolutionalNeural Network (CNN), Recurrent Neural Network (RNN), Deep BeliefNetwork (DBN), etc.

Deep learning will be described in more detail with reference to FIGS. 9to 12.

The object recognition module 144 may determine an attribute of anobject in input image data based on weights between nodes included inthe DNN.

Meanwhile, when the sensor unit 170 senses an object while the mobilerobot 100, 100 a, or 100 b moves, the controller 140 may perform controlto extract a part from an image acquired by the image acquisition unit120 in correspondence to a direction in which the object sensed by thesensor unit 170 is present.

The image acquisition unit 120, especially the front camera 120 a, mayacquire an image within a predetermined range of angles in a movingdirection of the mobile robot 100, 100 a, or 100 b.

The controller 140 may distinguish an attribute of the object present inthe moving direction, not by using the whole image acquired by the imageacquisition unit, especially the front camera 120 a, but by using only apart of the image.

Referring to FIG. 6, the controller 140 may further include an imageprocessing module 145 for extracting a part of an image acquired by theimage acquisition unit 120 in correspondence to a direction in which anobject sensed by the sensor unit 170 is present.

Alternatively, referring to FIG. 7, the mobile robot 100 b may furtherinclude an additional image processing unit 125 for extracting a part ofan image acquired by the image acquisition unit 120 in correspondence toa direction in which an object sensed by the sensor unit 170 is present.

The mobile robot 100 a in FIG. 6 and the mobile robot 100 b in FIG. 7are identical to each other, except for the image processing module 145and the image processing unit 125.

Alternatively, in some implementations, the image acquisition unit 120may extract a part of the image on its own, unlike the examples of FIGS.6 and 7.

The object recognition module 144 having trained through machinelearning has a higher recognition rate if a learned object occupies alarger portion of input image data.

Accordingly, the present invention extracts a different area of an imageacquired by the image acquisition unit 120 depending on a direction inwhich an object sensed by the sensor unit 170 such as an ultrasonicsensor, and use the extracted part of the image as data for recognition,thereby enhance a recognition rate.

The object recognition module 144 may recognize an object in theextracted part of the image, based on data that is pre-learned throughmachine learning.

In addition, the travel control module 141 may control driving of thetravel drive unit 160 based on an attribute of the recognized object.

Meanwhile, when the object is sensed from a forward-right direction ofthe front surface of the main body, the controller 140 may performcontrol to extract a right lower area of the image acquired by the imageacquisition unit; when the object is sensed from a forward-leftdirection of the main body, the controller 140 may perform control toextract a left lower area of the image acquired by the image acquisitionunit; and when the object is sensed from a forward direction of the mainbody, the controller 140 may perform control to extract a central lowerarea of the image acquired by the image acquisition unit.

In addition, the controller 140 may perform control to shift anextraction target area of an image acquired by the image acquisitionunit to correspond to a direction in which the sensed object is present,and then extract the extraction target area.

A feature of cropping a part from an image acquired by the imageacquisition unit 120 and extracting the cropped part will be describedin more detail with reference to FIGS. 15 to 21.

In addition, when the sensor unit 170 senses an object while the mobilerobot 100, 100 a, or 100 b moves, the controller 140 may perform controlto select, based on a moving direction and a moving speed of the mainbody 110, an image acquired at a specific point in time earlier than anobject sensing time by the sensor unit 170 from among a plurality ofcontinuous images acquired by the image acquisition unit 120.

In the case where the image acquisition unit 120 acquires an image usingan object sensing time of the sensor unit 170 as a trigger signal, anobject may not be included in the acquired image or may be captured in avery small size because the mobile robot keeps moving.

Thus, in one embodiment of the present invention, it is possible toselect, based on a moving direction and a moving speed of the main body110, an image acquired at a specific point in time earlier than anobject sensing time of the sensor unit 170 from among a plurality ofcontinuous images acquired by the image acquisition unit 120 and use theselected as data for object recognition.

Meanwhile, the object recognition module 144 may recognize an attributeof an object included in the selected image acquired at thepredetermined point in time, based on data that is pre-learned throughmachine learning.

Meanwhile, input data for distinguishing an attribute of an object, anddata for training the DNN may be stored in the storage 150.

An original image acquired by the image acquisition unit 120, andextracted images of a predetermined area may be stored in the storage150.

In addition, in some implementations, weights and biases of the DNN maybe stored in the storage 150.

Alternatively, in some implementations, weights and biases of the DNNmay be stored in an embedded memory of the object recognition module144.

Meanwhile, whenever a part of an image acquired by the image acquisitionunit 120 is extracted, the object recognition module 144 may perform alearning process using the extracted part of the image as training data,or, after a predetermined number of images is extracted, the objectrecognition module 144 may perform a learning process.

That is, whenever an object is recognized, the object recognition module144 may add a recognition result to update a DNN architecture, such as aweight, or, when a predetermined number of training data is secured, theobject recognition module 144 may perform a learning process using thesecured data sets as training data so as to update an DNN architecture.

Alternatively, the mobile robot 100, 100 a, or 100 b transmits theextracted part of the image to a predetermined server through thecommunication unit 190, and receive machine learning-related data fromthe predetermined server.

In this case, the mobile robot 100, 100 a, or 100 b may update theobject recognition module 141 based on the machine learning-related datareceived from the predetermined server.

FIG. 8 is an exemplary schematic internal block diagram of a serveraccording to an embodiment of the present invention.

Referring to FIG. 8, a server 70 may include a communication unit 820, astorage 830, a learning module 840, and a processor 810.

The processor 810 may control overall operations of the server 70.

Meanwhile, the server 70 may be a server operated by a manufacturer of ahome appliance such as the mobile robot 100, 100 a, or 100 b, a serveroperated by a service provider, or a kind of Cloud server.

The communication unit 820 may receive a diversity of data, such asstate information, operation information, controlled information, etc.,from a mobile terminal, a home appliance such as the mobile robot 100,100 a, or 100 b, etc., a gate way, or the like.

In addition, the communication unit 820 may transmit data responsive tothe received diversity of data to a mobile terminal, a home appliancesuch as the mobile robot 100, 100 a, 100 b, etc., a gate way or thelike.

To this end, the communication unit 820 may include one or morecommunication modules such as an Internet module, a communicationmodule, etc.

The storage 830 may store received information and data necessary togenerate result information responsive to the received information.

In addition, the storage 830 may store data used for learning machine,result data, etc.

The learning module 840 may serve as a learning machine of a homeappliance such as the mobile robot 100, 100 a, or 100 b, etc.

The learning module 840 may include an artificial network, for example,a Deep Neural Network (DNN) such as a Convolutional Neural Network(CNN), a Recurrent Neural Network (RNN), a Deep Belief Network (DBN),etc., and may train the DNN.

As a learning scheme for the learning module 840, both unsupervisedlearning and supervised learning may be used.

Meanwhile, according to a setting, the controller 810 may performcontrol to update an artificial network architecture of a home appliancesuch as the mobile robot 100, 100 a, or 100 b, into a learned artificialneural network architecture.

FIGS. 9 to 12 are diagrams for explanation of deep learning.

Deep learning is a kind of machine learning, and goes deep inside datain multiple stages.

The deep learning may represent a set of machine learning algorithmsthat extract more critical data from a plurality of data sets on eachlayer than on the previous layer.

Deep learning architectures may include an ANN and may be, for example,composed of a DNN such as a Convolutional Neural Network (CNN_, aRecurrent Neural Network (RNN), a Deep Belief Network (DBN), etc.

Referring to FIG. 9, the ANN may include an input layer, a hidden layer,and an output layer. Each layer includes a plurality of nodes, and eachlayer is connected to a subsequent layer. Nodes between adjacent layersmay be connected to each other with having weights.

Referring to FIG. 10, a computer (machine) constructs a feature map bydiscovering a predetermined pattern from input data 1010. The computer(machine) may recognize an object by extracting a low level feature 1020through an intermediate level feature 1030 to an upper level feature1040, and output a corresponding result 1050.

The ANN may abstract a higher level feature on a further subsequentlayer.

Referring to FIGS. 9 and 10, each node may operate based on anactivation model, and an output value corresponding to an input valuemay be determined according to the activation model.

An output value of a random node, for example, the low level feature1020, may be input to a subsequent layer connected to the correspondingnode, for example, a node of the intermediate level feature 1030. Thenode of the subsequent layer, for example, the node of the intermediatelevel feature 1030, may receive values output from a plurality of nodesof the lower level feature 1020.

At this time, a value input to each node may be an output value of aprevious layer with a weight applied. A weight may refer to strength ofconnection between nodes.

In addition, a deep learning process may be considered a process fordetermining an appropriate weight.

Meanwhile, an output value of a random node, for example, theintermediate level feature 1030, may be input to a node of a subsequentlayer connected to the corresponding node, for example, a node of thehigher level feature 1040. The node of the subsequent layer, forexample, the node of the higher level feature 1040, may receive valuesoutput from a plurality of nodes of the intermediate level feature 1030.

The ANN may extract feature information corresponding to each levelusing a trained layer corresponding to each level. The ANN may performabstraction sequentially such that a predetermined object may berecognized using feature information of the highest layer.

For example, face recognition with deep learning is as follows: acomputer may classify bright pixels and dark pixels by degrees ofbrightness from an input image, distinguish simple forms such asboundaries and edges, and then distinguish more complex forms andobjects. Lastly, the computer may recognize a form that defines a humanface.

A deep learning architecture according to the present invention mayemploy a variety of well-known architectures. For example, the deeplearning architecture may be a CNN, RNN, DBN, etc.

The RNN is widely used in processing natural languages and efficient inprocessing time-series data, which changes over time, and thus, the RNNmay construct an artificial neural network architecture by piling uplayers each time.

The DBN is a deep learning architecture that is constructed by piling upmultiple layers of Restricted Boltzman Machine (RBM) which is a deeplearning technique. If a predetermined number of layers is constructedby repeating RBM learning, a DBN having the predetermined number oflayers may be constructed.

The CNN is an architecture widely used especially in object recognitionand the CNN will be described with reference to FIGS. 11 and 12.

The CNN is a model that imitates human brain functions based on anassumption that a human extracts basic features of an object, performscomplex calculation on the extracted basic features, and recognizes theobject based on a result of the calculation.

FIG. 11 is a diagram of a CNN architecture.

A CNN may also include an input layer, a hidden layer, and an outputlayer.

A predetermined image is input to the input layer.

Referring to FIG. 11, the hidden layer may be composed of a plurality oflayers and include a convolutional layer and a sub-sampling layer.

In the CNN, a pooling or non-linear activation function is used to addnon-linear features and various filters so as to basically extractfeatures of an image through convolution computation.

Convolution is mainly used in filter computation in image processingfields and used to implement a filter for extracting a feature from animage.

For example, as shown in FIG. 12, if convolution computation isrepeatedly performed with respect to the entire image by moving a 3×3window, an appropriate result may be obtained according to a weight ofthe window.

Referring to (a) of FIG. 12, if convolution computation is performedwith respect to a predetermined area 1210 using a 3×3 window, a result1201 is outcome.

Referring to (b) of FIG. 12, if computation is performed again withrespect to an area 1220 by moving the 3×3 window to the right, apredetermined result 1202 is outcome.

That is, as shown in (c) of FIG. 12, if computation is performed withrespect to an entire image by moving a predetermined window, a finalresult may be obtained.

A convolution layer may be used to perform convolution filtering forfiltering information extracted from a previous layer with a filter of apredetermined size (for example, a 3×3 window exemplified in FIG. 12).

The convolutional layer performs convolutional computation on inputimage data 1100 and 1102 with a convolutional filter, and generatesfeature maps 1101 and 1103 representing features of an input image 1100.

As a result of convolution filtering, filtered images in the same numberof filters included in the convolution layer may be generated. Theconvolution layer may be composed of nodes included in the filteredimages.

In addition, a sub-sampling layer paired with a convolution layer mayinclude feature maps in the same number as feature maps of the pairedconvolutional layer.

The sub-sampling layer reduces dimension of the feature maps 1101 and1103 through sampling or pooling.

The output layer recognizes the input image 1100 by combining variousfeatures represented in a feature map 1104.

An object recognition module of a mobile robot according to the presentinvention may employ the aforementioned various deep learningarchitectures. For example, a CNN widely used in recognizing an objectin an image may be employed, but aspects of the present invention arenot limited thereto.

Meanwhile, training of an ANN may be performed by adjusting a weight ofa connection line between nodes so that a desired output is output inresponse to a given input. In addition, the ANN may constantly updatethe weight due to the training. In addition, back propagation or thelike may be used to train the ANN.

FIGS. 13 and 14 are flowcharts illustrating a method for controlling amobile robot according to an embodiment of the present invention.

Referring to FIGS. 2 to 7 and 13, the mobile robot 100, 100 a, or 100 bmay perform cleaning while moving according to a command or a setting(S1310).

The sensor unit 170 includes an obstacle detection sensor 131 and maysense a forward object.

When an object is sensed by the sensor unit 170 while the mobile robotmoves (S1320), the image acquisition unit 120 may acquire an image of anarea forward of the main body 110 (S1330).

In this case, the controller 140 may perform control to crop and extractpart of an image acquired by the image acquisition unit 120 incorrespondence to a direction, in which the object sensed by the sensor170 is present (S1340).

In some implementations, the image processing module 145 of thecontroller 140 may extract a part of the image. Alternatively, thecontroller 140 may perform control such that the image processing unit125 additionally provided extracts a part of the image. Alternatively,the controller 140 may perform control such that the image acquisitionunit 120 extracts a part of the image.

Meanwhile, the obstacle recognition module 144 may recognize an objectbased on data, which is pre-learned through machine learning, in theextracted part of the image (S1350).

The object recognition module 144 may include an ANN having trained torecognize attributes such as types of objects through machine learning,and may recognize an attribute of an object based on pre-learned data(S1350).

For example, a CNN, which is one of deep learning architectures, may beembedded in the object recognition module 144, and a pretrained CNN mayrecognize an attribute of an object included in input data and output aresult of the recognition.

The object recognition module 144 having trained through machinelearning has a higher recognition rate if a learned object occupies alarger portion of input image data.

Accordingly, the present invention may extract a different part of animage acquired by the image acquisition unit 120 depending on adirection in which an object sensed by the sensor unit 170 such as anultrasonic sensor, and use the extracted part of the image as data forrecognition, thereby enhancing a recognition rate.

The feature of cropping and extracting a part of an image acquired bythe image acquisition unit 120 will be described in more detail withreference to FIGS. 15 to 21.

Meanwhile, the travel control module 141 may control driving of thetravel drive unit 160 based on the recognized attribute of the object(S1360).

For example, if the recognized object is an object having a height thatthe mobile robot 100 is not able to climb, the travel control module 141may perform control to travel while detouring around the object.

In addition, if the recognized object is an object, such as a low bump,having a height that the mobile robot 100 is able to climb, such as alow door threshold, the travel control module 141 may perform control tokeep traveling in a straight forward direction.

In addition, if an object is recognized as an object that can restrictedthe mobile robot 100 during traveling even though the object is a lowheight object such as a fan base, a human hair, a multi plug, anelectric wire, etc., the travel control module 141 may perform controlto travel while detouring around the object.

Meanwhile, by performing the aforementioned object recognition processrepeatedly and achieving a final recognition result based on a pluralityof recognition results, it is possible to further enhance recognitionaccuracy.

For example, the controller 140 may store position information of asensed object and position information of the mobile robot in thestorage 150, and perform control such that an area of a predeterminedsize around the position of the sensed object is registered as an objectarea in a map.

Thereafter, whenever the mobile robot passes the registered object area,an attribute of the object may be recognized, and a final attribute ofthe object may be determined based on the plurality of recognitionresults.

In addition, if an object is recognized as an object that can confinethe mobile robot 100 during traveling even though the object is a lowheight object such as a fan base, a human hair, a multi plug, anelectric wire, etc., the travel control module 141 may perform controlto travel while detouring around the object. This embodiment will bedescribed in more detail with reference to FIGS. 29 to 34.

Referring to FIGS. 2 to 7 and 14, the mobile robot 100, 100 a, or 100 bmay move according to a command or a setting (S1410).

In the case where the sensor unit 170 includes ultrasonic sensor, thesensor unit 170 may sense an object in response to sensing of areflected ultrasonic signal (S1420).

Meanwhile, the image acquisition unit 120 may acquire images byconstantly photographing the front area or the surroundings of themobile robot 100, 100 a, or 100 b or by photographing the front area orthe surroundings of the mobile robot 100, 100 a, or 100 b in response tosensing an obstacle by the sensor unit 170.

The controller 140 may perform control to extract a part of an imageacquired by the image acquisition unit, the part which corresponds to anarea where the ultrasonic signal is sensed (S1430). For example, if thearea where the ultrasonic signal is sensed is a front right area, thecontroller 140 may perform control to extract a left area in an areaacquired by the image acquisition unit 120. In addition, if the areawhere the ultrasonic signal is sensed is a front central area, thecontroller 140 may perform control to extract a central area in an imageacquired by the image acquisition unit 120.

Meanwhile, based on data pre-learned through machine learning, thecontroller 140 may determine an attribute of an object sensed in theextracted part of the image.

In addition, the controller 140 may determine whether the sensed objecthas a height that the mobile robot 100 is able to climb (S1440).

If the recognized object is an object having a height that the mobilerobot is not able to climb (S1440), the controller 140 may performcontrol to travel while detouring around the object by rotating by 90degrees (S1455).

Meanwhile, if the recognized object is an object having a height thatthe mobile robot is able to climb (S1440), the controller 140 maydetermine attribute information of the sensed object (S1445). That is,the controller 140 may determine whether the recognized object is lesslikely to restrict the mobile robot and thus considered an object thatthe robot cleaner is able to climb.

If the recognized object is determined as an object that the mobilerobot is able to climb, the controller 140 may perform control to keepmoving straight forward.

Conventionally, whether a sensed object has a height that a mobile robotis able to climb is determined, and, when the sensed object is an objecthaving a low height, the mobile robot moves straight forward.

However, when an object such as an electric wire is present, a mobilerobot is sometimes restricted and thus restricted by the electric wire.

In addition, the mobile robot may try to escape from the restrictedstate by performing a motion of shaking to the left/right, but this maylead to an accident, for example, peeling off of the electric wire.

However, the present invention may enhance reliability by recognizingobject attribute information with learning machine and image informationand by determining a travel pattern depending on a recognized objectattribute.

FIGS. 15 to 21 are diagrams for explanation of a method for controllinga mobile robot according to an embodiment of the present invention,especially examples of cropping and extracting a part of an imageacquired by the image acquisition unit 120.

Referring to FIG. 15, the mobile robot 100 may sense an obstacle 1500,such as a fan, using the sensor unit 170 while moving. FIG. 15illustrates an example in which the obstacle 1500 is sensed from acenter in a moving direction of the mobile robot 100.

When the mobile robot 100 senses the obstacle 1500 while moving, theimage acquisition unit 120 may photograph the front area of the mobilerobot 100 and acquire an image 1600 including at least part of theobject 1500, as shown in (a) of FIG. 16.

The controller 140 may perform control to extract a part 1610 of theimage 1600 acquired by the image acquisition unit 120 in correspondenceto a direction in which the object 1500 sensed by the sensor unit 170 ispresent.

Referring to FIGS. 15 and 16, when the object 1500 is sensed from aforward direction of the main body 110, the controller 140 may performcontrol to extract a central lower area 1610 from an image acquired bythe image acquisition unit 120.

Meanwhile, the controller 140 may use an extracted image 1620, as shownin (b) of FIG. 16, as input data for object attribute recognition.

Based on data that is pre-learned through machine learning, the objectrecognition module 144 may recognize that the object 1500 is a fan.Referring to FIG. 17, the image acquisition unit 120 may acquire aplurality of images by continuously photographing while the mobile robot100 travels. The mobile robot 100 may acquire a first image 1711 at afirst position 1710, a second image 1721 at a second position 1720, anda third image 1731 at a third position 1730.

Meanwhile, when the sensed object is not an object that the mobile robotis able to climb, the travel control module 141 may control the traveldrive unit 160 to perform avoidance operation such as proceeding afterrotation.

Meanwhile, the extracted image 1620 may be stored in the storage 150. Inaddition, the original image 1600 acquired by the image acquisition unit120 may be stored in the storage 150.

The extracted image 1620 stored in the storage 150 may be used astraining data.

Meanwhile, when an object is sensed from a forward-right direction ofthe main body 119, the controller 140 may perform control to extract aright lower area from an image acquired by the image acquisition unit120, and, when an object is sensed from a forward-left direction of themain body 110, the controller 140 may perform control to extract a leftlower area from an image acquired by the image acquisition unit 120.

FIG. 17 shows an example in which the object 1500 is sensed from a rightside in a moving direction of the mobile robot 100.

Referring to (a) of FIG. 18, when the object 1500 is sensed from aforward-right direction of the main body 110, the controller 140 mayperform control to extract a right lower area from an image acquired bythe image acquisition unit 120.

The controller 140 may use an extracted image 1820, as shown in (b) ofFIG. 18, as input data to recognize an attribute of the object.

The present invention may extract a central, left, or right area of animage based on a direction in which an object is sensed, rather thancropping an area in a preset size around a central area of the entireimage.

Accordingly, objects as many as possible may be included in input datafor recognition. Since a machine recognizes what occupies the largestproportion of an image, it is possible to enhance a rate of recognitionof an object attribute.

Meanwhile, since the mobile robot 100 travels on a floor in apredetermined space, a lower area may be extracted from an acquiredimage so that more objects may be included in input data for therecognition.

Meanwhile, referring to (a) of FIG. 19, (a) of FIG. 20, and (a) of FIG.21, the sensor unit 170 according to an embodiment of the presentinvention may include a first sensor 51 disposed on the front surface ofthe main body of the mobile robot, and a second sensor S2 and a thirdsensor S3 respectively spaced apart from the first sensor 51 in the leftand right direction.

In this case, the first sensor 51 may operate as a transmitter, and thesecond sensor S2 and the third sensor S3 may operate as a receiver. Forexample, the first sensor 51 may transmit an ultrasonic signal, and thesecond sensor S2 and the third sensor S3 may receive a signal reflectedby an object. If the signal reflected by the object is received, it ispossible to determine a direction in which the object is present and adistance to the object with well-known recognition methods usingultrasound.

The drawing (a) of FIG. 19 shows an example in which an object X1 issensed at the center from the forward direction of the mobile robot 100.If a distance L1 from the sensed object X1 to the second sensor S2 and adistance L2 from the sensed object X1 to the third sensor S3 are equal(L1=L2), it is possible to determine that the object X1 is sensed at thecenter from the forward direction of the mobile robot.

In this case, as show in (b) of FIG. 19, it is possible to extract apredetermined area 1910 of size a2×b2 from the central lower side of anoriginal image 1900 of size a1×b1 acquired by the image acquisition unit120.

Meanwhile, the controller 140 may perform control to extract anextraction target area from an image acquired by the image acquisitionunit 120 by shifting the extraction target area in proportional to adifference between a distance L1 from the sensed object X1 to the secondsensor S2 and a distance L2 from the sensed object X1 and the thirdsensor S3.

(a) of FIG. 20 illustrates an example in which an object X1 is sensedfrom a forward-right direction of the mobile robot 100. If the distanceL1 from the sensed object X1 to the second sensor S2 is greater than thedistance L2 from the sensed object X1 to the third sensor S3 (L1>L2), itis possible to determine that the object is sensed from theforward-right direction of the mobile robot 100.

In this case, as shown in (b) of FIG. 20, it is possible to extract apredetermined area 1920 of size a2×b2 from the right lower side of anoriginal image 1900 of size a1×b1 acquired by the image acquisition unit120.

In comparison between (b) of FIG. 19 and (b) of FIG. 20, it is foundthat a central point CP2 and a start point SP2 of the extraction targetarea 1920 of when the object X1 is sensed from the forward-rightdirection of the mobile robot 100 have been shifted by a predeterminedvalue d1 in a right direction from a central point CP1 and a start pointSP1 of the extraction target area 1910 of when the object X1 is sensedat the center from the forward direction of the mobile robot 100.

In this case, shifting by the predetermined value d1 may be proportionalto the difference (L1−L2) between the distance L1 from the sensed objectX1 to the second sensor S2 and the distance L2 from the sensed object X1to the third sensor S3.

(a) of FIG. 21 illustrates an example in which an object X1 is sensedfrom a forward-left direction of the mobile robot 100. If the distanceL1 from the sensed object X1 to the second sensor S2 is smaller than thedistance L2 from the sensed object X1 to the third sensor S3 (L1<L2), itis possible to determine that the object X1 is sensed in theforward-left direction of the mobile robot 100.

In this case, as shown in (b) of FIG. 21, it is possible to extract apredetermined area 1930 of size a2×b2 from the left lower side of anoriginal image 1900 of size a1×b1 acquired by the image acquisition unit120.

In comparison between (b) of FIG. 19 and (b) of FIG. 21, it is foundthat a central point CP3 and a start point SP3 of the extraction targetarea 1930 of when the object X1 is sensed from the forward-leftdirection of the mobile robot 100 have been shifted by a predeterminedvalue d2 in the left direction from a central point CP1 and a startpoint SP1 of the extraction target area 1910 of when the object X1 issensed at the center from the forward direction of the mobile robot 100.

In this case, shifting by the predetermined value d2 may be proportionalto the difference (L2−L1) between the distance L1 from the sensed objectX1 to the second sensor S2 and the distance L2 from the sensed object X1to the third sensor S3.

Meanwhile, as the mobile robot according to an embodiment of the presentinvention performs a learning process with training data which is theextracted image, the mobile robot may constantly update architectures ofan ANN and a DNN.

Alternatively, the mobile robot may transmit the extracted image to apredetermined server, and receive machine learning-related data from thepredetermined server. Thereafter, the mobile robot may update the objectrecognition module based on the machine learning-related data receivedfrom the predetermined server.

FIG. 22 is a diagram for explanation of an operation method of a mobilerobot and a server according to an embodiment of the present invention,and FIG. 23 is a flowchart of an operation method of a mobile robot anda server according to an embodiment of the present invention.

Referring to FIGS. 22 and 23, a DNN architecture 144 a, such as a CNN,may be embedded in the object recognition module 144 of the mobile robot100.

A pre-learned DNN architecture 144 a may receive input data forrecognition (S2310), recognize an attribute of an object included in theinput data (S2320), and output a result of the recognition (S2330).

Unknown data unrecognizable by the DNN architecture 144 a may be storedin the storage 150 or a storage space 144 b within the objectrecognition module 144 (S2340).

Meanwhile, the unknown unrecognizable by the object recognition module144 may be transmitted to the server 70 through the communication unit190 (S2341). In addition, data recognized by the object recognitionmodule 144 may be transmitted to the server 70 as well.

The server 70 may generate configuration of learned weights, and theserver 70 may learn the DNN architecture with training data.

The server 70 may train the DNN based on the received data (S2342), andupdate the DNN by transmitting updated DNN architecture data to themobile robot 100 (S2343).

FIGS. 24 to 26 are diagrams for explanation of a method for controllinga mobile robot according to an embodiment of the present invention.

The mobile robot according to an embodiment of the present invention maydivide an object sensing area into three divided areas such as thecenter, left, and right areas or into a more number of divided areas.

FIG. 24 illustrates an example in which an object is sensed with sixdivided areas.

Referring to (a) of FIG. 24, an object may be sensed by dividing anobject sensing area 2400 into a central upper area CU, a central lowerarea CD, a right upper area RU, a right lower area RD, a left upper areaRU, and a left lower area LD.

In addition, the mobile robot according to an embodiment of the presentinvention may extract a part of an image acquired by the imageacquisition unit 120 in correspondence to an area where an object issensed.

Referring to (b) of FIG. 24, an object 2410 may be sensed over thecentral upper area CU and the right upper area RU. In this case, it ispossible to crop the central upper area and the right upper area fromthe entire image acquired by the image acquisition unit 120.

Referring to (c) of FIG. 24, an object 2420 may be sensed over thecentral lower area CD, the right lower area RD, and the left lower areaLD. In this case, it is possible to crop the lower areas CD, RD, and LDfrom the entire image acquired by the image acquisition unit 120.

Alternatively, it is possible to further sub-divide the obstacle sensingarea and crop a sub-divided area.

For example, referring to (c) of FIG. 24, in the case where the object2420 is sensed over the central lower area CD, the right lower area RD,and the left lower area LD, it is possible to crop a lower area 2325 inthe central lower area CD, the right lower area RD, and the left lowerarea LD.

FIGS. 25 and 26 are diagrams for explanation of object recognition bythe object recognition module 144.

Referring to FIG. 25, the object recognition module 144 may classifyobjects into classes such as a fan, a home theater, a multi plug, a lampbase, a human hair, a bump, etc., and recognize the objects in adistinguishable manner.

In addition, the object recognition module 144 may classify the classessuch as a fan, a home theater, a multi plug, a lamp base, a human hair,etc. as a dangerous object super-class which is a higher conceptuallevel, and may recognize the object in a distinguishable manner.

In addition, the object recognition module 144 may classify an object,such as a bump and the like, which the mobile robot is able to travelstraight forward, into a non-dangerous object super-class and recognizethe object in a distinguishable manner.

Referring to (a) of FIG. 26, the object recognition module 144 mayrecognize an input image and obtain a recognition result indicative of afan with a confidence of 0.95 and a recognition result indicative of ahome theater with a confidence of 0.7. In this case, the objectrecognition module 144 may output the fan, which is a recognition resulthaving a higher confidence of 0.7, as a recognition result with respectto the input image.

Meanwhile, a confidence may be normalized within a range from 0.0 to1.0.

Referring to (b) of FIG. 26, the object recognition module 144 mayrecognize an input image and obtain a recognition result indicative of afan with a confidence of 0.35 and a recognition result indicative of ahome theater with a confidence of 0.4.

For example, in the case where it is set to ignore a confidence of 0.6or less, the object recognition module 144 may not select any specificrecognition result because the two recognition results are lower than areference threshold, and the object recognition module 144 may determineunknown data.

Referring to (c) of FIG. 26, the object recognition module 144 mayrecognize an input image and obtain a recognition result indicative of afan with a confidence of 0.95 and a recognition result indicative of ahome theater with a confidence of 0.9.

For example, in the case where it is set to select a recognition resultwith a confidence of 0.9 or higher as a final recognition result, theobject recognition module 144 may not select any specific recognitionresult because the two recognition results are higher than a referencethreshold, and thus, the object recognition module 144 may determine anobject as a dangerous object which is a higher conceptual level.

Alternatively, in the case where it is set to confirm a higherrecognition result when a difference between confidences is 0.15 orhigher, the object may be determined as a dangerous object that is ahigher conceptual level.

Meanwhile, even in the case where an object is determined as a dangerousobject, the travel control module 141 may control the travel drive unit160 so as to move while detouring around the dangerous object.

FIG. 27 is a flowchart of a method for controlling a mobile robotaccording to an embodiment of the present invention.

Referring to FIG. 27, firstly, the mobile robot 100, 100 a, or 100 b mayperform cleaning while traveling according to a command or a setting(S2710).

Meanwhile, the image acquisition unit 120 may acquire a plurality ofimages by continuously photographing the surroundings of the main body110 during traveling (S2720). The plurality of continuous imagesacquired by the image acquisition unit 120 may be stored in the storage150.

If an object is sensed by the sensor unit 170 during traveling (S2730),the controller 140 may select an image acquired at a specific point intime earlier than an object sensing time of the sensor unit 170 fromamong the plurality of continuous images based on a moving direction anda moving speed of the main body 110 (S2740).

The moving direction and the moving speed may be calculated by thetravel control module 141 or the like based on an output from anoperation detection sensor of the sensor unit 170.

Meanwhile, in the case where the mobile robot is set to travel at areference speed in normal travel, the moving speed is constant, andthus, it is possible to select an image acquired at a specific point intime by determining a moving direction of the main body 110.

Meanwhile, in the case where the moving direction corresponds to forwardtravel or rotation by an angle smaller than a predetermined referencethreshold (reference angle), the controller 140 may select an imageacquired at a specific point in time earlier than the object sensingtime of the sensor unit 170 from the plurality of continuous images.

In general, a sensing range of a sensor such as an ultrasonic sensor isgreater than a photographing range of a camera.

Accordingly, in the case where an image is acquired in response to asignal indicative of sensing of an object by the sensor unit 170, themobile robot is still traveling and thus the acquired image may notinclude features of the object. In addition, in the case where a sensingrange of the sensor unit 170 is short, an acquired image is more likelynot to include features of an object.

Accordingly, in the case of straight forward traveling or approximatelystraight forward traveling, the controller 140 may select an imageacquired at a specific point in time earlier than the object sensingtime of the sensor unit 170 based on a moving direction and a movingspeed.

In this case, if the moving speed is slower, the controller 140 mayselect an image acquired at a point in time further earlier than theobject sensing time of the sensor unit 170.

If the moving speed is faster, the mobile robot 100 travels a longerdistance after the object sensing time of the sensing time 170. In otherwords, when the image acquisition unit 120 acquires a plurality ofimages at a constant speed, the mobile robot travels a longer distanceduring a period of time between a point in time to capture a specificframe and a point in time to capture a subsequent frame.

Therefore, as a moving speed is faster the probability that an areaoccupied by an object in an image is large is higher when the image atthe point in time closer to an object sensing time of the sensor unit170 is selected.

In contrast, as a moving speed decreases, the image acquisition unit 120acquires more images while traveling the same distance, and thus, it maybe preferable to select an image captured at a point in time furtherearlier than an object sensing time of the sensor unit 170.

The object recognition module 144 may select the image acquired at thespecific point in time, and use the image as input data for recognitionof the object.

Meanwhile, the object recognition module 144 trained through the machinelearning exhibits a higher recognition rate if a learned object occupiesa higher proportion of input image data.

Thus, the controller 140 may perform control to crop a part of theselected image acquired at the specific point in time in correspondenceto a direction in which an object sensed by the sensor unit 170 ispresent, and to extract the cropped part (S2750).

As a different part is extracted from an image acquired by the imageacquisition unit 120 depending on a direction in which an object sensedby the sensor unit 170 is present, and the extracted part of the imageis used as data for recognition, a recognition rate may be enhanced.

The object recognition module 144 may recognize an attribute of theobject included in the image of the part that is extracted from theselected image acquired at the specific point in time (S2760).

Based on data pre-learned through machine learning, the objectrecognition module 144 may recognize an attribute of the object includedin the image of the part that is extracted from the selected imageacquired at the specific point in time.

The object recognition module 144 may include an ANN trained torecognize attributes such as types of objects through machine learning,and recognize an attribute of an object based on pre-learned data(S2760).

For example, a CNN, which is one of deep learning architectures, may beembedded in the object recognition module 144, and a pre-trained CNN mayrecognize an attribute of an object included in input data and output aresult of the recognition.

Meanwhile, the travel control module 141 may control the travel driveunit 160 based on the recognized attribute of the object (S2770).

For example, if the recognized object is an object having a height thatthe mobile robot 100 is not able to climb, the travel control module 141may perform control to travel while detouring around the object.

In addition, if the recognized object is an object, such as a low bump,having a height that the mobile robot 100 is able to climb, such as alow door threshold, the travel control module 141 may perform control tokeep traveling in a straight forward direction.

In addition, if an object is recognized as an object that can restrictthe mobile robot 100 during traveling even though the object is a lowheight object such as a fan base, a human hair, a multi plug, anelectric wire, etc., the travel control module 141 may perform controlto travel while detouring around the object.

FIG. 28 is a diagram for explanation of a method for controlling amobile robot according to an embodiment of the present invention,especially an example in which an object 2800 is sensed from a forwarddirection of the mobile robot 100.

Referring to FIG. 28, the image acquisition unit 120 may acquire aplurality of images by continuously photographing while the mobile robot100 travels. The mobile robot 100 may acquire a first image 2811 at afirst position 2810, a second image 2821 at a second position 2820, anda third image 2831 at a third position 2830.

A predetermined number of images may be stored in the storage 150. Inaddition, if a preset number of images is acquired, an image acquired atthe earliest point in time may be deleted and a newly acquired image maybe stored.

The mobile robot 100 may start an image recognition processing inaccordance with a trigger signal responsive to sensing an ultrasonicsignal.

However, since an ultrasonic sensor has a short range, a feature of arecognition subject may not be found in the image 2831 acquired when theobject 2800 is sensed and triggered.

Thus, the present invention may store continuous images in the storage150, determine whether a direction of travel is a straight forwarddirection, and perform an object recognition process not using the thirdimage 2831 captured at the trigger event, but using the first image2811.

Since the mobile robot 100 travels at a constant speed most of times, itis possible to perform an object recognition process simply bydetermining whether the mobile robot 100 is traveling straight forward,and by selecting an image acquired at a specific point in time, forexample, a previous image two frames earlier than an object sensing timeof the sensor unit 170.

In addition, by taking into consideration even a sensing range of thesensor unit 170, performance, and a processing speed of the objectrecognition process, the controller 140 may determine how much earlier aprevious image is selected compared to the object sensing time of thesensor unit 170.

Meanwhile, in some embodiments, by using only a part of the imageacquired at the selected point in time, rather than the entire image, asinput data for object recognition, it is possible to enhance arecognition rate.

In this embodiment, it is possible to extract the central area, the leftarea, and the right area of an image based on a direction in which anobject is sensed, rather than cropping an area of a predetermined sizearound the center area of the image.

For example, when the object is sensed from a forward-right direction ofthe main body, a right lower area of the image acquired at the selectedpoint in time may be extracted; when the object is sensed from aforward-left direction of the main body, a left lower area of the imageacquired at the selected point in time may be extracted; and when theobject is sensed from a forward direction of the main body, a centrallower area of the image acquired at the selected point in time may beextracted

Accordingly, objects as many as possible may be included in input datafor recognition. Since a machine recognizes what occupies the largestproportion of an image, it is possible to enhance a rate of recognitionof an object attribute.

Meanwhile, by performing the aforementioned object recognition processrepeatedly and achieving a final recognition result based on a pluralityof recognition results, it is possible to further enhance recognitionaccuracy.

For example, the controller 140 may store position information of asensed object and position information of the mobile robot in thestorage 150, and perform control such that an area of a predeterminedsize around the position of the sensed object is registered as an objectarea in a map.

Thereafter, whenever the mobile robot passes the registered object area,an attribute of the object may be recognized and a final attribute ofthe object may be determined based on the plurality of recognitionresults.

The controller 140 may sequentially recognize an attribute of an objectin a predetermined object area with respect to images acquired by theimage acquisition unit 120, and determine a final attribute of theobject based on such a plurality of recognition results from thesequential recognition.

FIG. 29 is a flowchart of a method for controlling a mobile robotaccording to an embodiment of the present invention.

Referring to FIG. 29, the mobile robot 100, 100 a, or 100 b may performcleaning while moving according to a command or a setting (S2910).

The sensor unit 170 may include an object detection sensor 131 and sensean object within a predetermined range in a forward direction of themoving direction. (S2920).

If an object is sensed by the sensor unit 170 (S2920) during movement,position information of the sensed object and position information ofthe mobile robot 100, 100 a, or 100 b may be stored in the storage 150(S2930).

In addition, an area having a predetermined size around the position ofthe sensed object may be registered as an object area in a map stored inthe storage 150 to store and manage the area (S2930).

The position information of the mobile robot may be calculated fromodometry information, and the odometry information may be constructedbased on data sensed by the aforementioned operation sensor of thesensor unit 170.

Meanwhile, the image acquisition unit 120 may acquire a plurality ofimages in the object area during the movement (S2940).

The image acquisition unit 120 may register the object area, and acquirea plurality of images by continuously photographing images in the objectarea before the mobile robot 100, 100 a, or 100 b departs out of theobject area.

In addition, the mobile robot 100, 100 a, or 100 b may acquire an imageof the entire object area while performing avoidance driving withrespect to the sensed object.

Meanwhile, photographing an object area while traveling in the objectarea in order to explore the entire object area may bring significantchange in a travel path of the mobile robot 100, 100 a, or 100 b.

Thus, it is preferable that the mobile robot 100, 100 a, or 100 bacquires images about an object by photographing the object areawhenever passing the object area in normal travel.

For example, suppose that the mobile robot 100, 100 a, or 100 b travelsin a basic zigzag pattern by moving in a straight forward direction,changing the moving direction, and then moving a distance in a directionopposite to the initial moving direction until sensing an object: inthis case, the mobile robot 100, 100 a, or 100 b may acquire a pluralityof images in a manner of acquiring a predetermined number of images byphotographing while passing an object area when first traveling straightforward, and in a manner of acquiring a predetermined number of imagesby photographing while passing the object area again when returning backto the initial position.

Meanwhile, the object recognition module 144 may recognize an attributeof an object, based on data which is pre-learned through learningmachine, with respect to images acquired by the image acquisition unit120 during movement in the object area (S2950).

The object recognition module 144 may include an ANN having trained torecognize attributes such as types of objects through machine learning,and may recognize an attribute of an object based on pre-learned data(S2950).

For example, a CNN, which is one of deep learning architectures, may beembedded in the object recognition module 144, and a pretrained CNN mayrecognize an attribute of an object included in input data and output aresult of the recognition.

After performing an object recognition process with respect to one imageacquired in the object area and then performing an object recognitionprocess with respect to another image, the object recognition module 144may select and store a more accurate recognition result from tworecognition results regarding the two images.

Meanwhile, the object recognition module 144 may compare confidencesincluded in the two recognition results to select a more accuraterecognition result. A result of the last recognition process may be thecurrent object recognition result, and, when a recognition process isperformed with respect to a subsequent image, the current objectrecognition result may become a previous object recognition result and aresult of the recognition process with respect to the subsequent imagemay become the current object recognition result.

Meanwhile, when sequentially recognizing an attribute of an object withrespect to images acquired by the image acquisition unit 120 (S2950),the object recognition module 144 may repeatedly compare the currentobject recognition result and a previous object recognition result.

Whenever passing the object area in normal travel, the mobile robot 100,100 a, or 100 b may acquire images by photographing the object area.

The object recognition module 144 may perform an image recognitionprocess with respect to acquired images in sequence, compare a result ofthe recognition with a previous object recognition result, and select arecognition result with a higher confidence.

Preferably, the object recognition module 144 may compare the currentobject recognition result and a previous object recognition result, and,if the current object recognition result and the previous objectrecognition result are identical, the object recognition module 144 maymaintain the current object recognition result and increase a confidenceby applying a predetermined weight.

That is, if the current object recognition result and the previousobject recognition result are identical, it is not necessary to changethe object recognition result, and therefore, it is possible to increasea confidence for the recognition result by applying the predeterminedweight, while maintaining the object recognition result.

Thus, whenever an identical recognition result is outcome, the objectrecognition module 144 may perform control such that a confidence isincreased by applying the predetermined weight and stored.

In addition, if the current object recognition result and the previousobject recognition result are not identical, the object recognitionmodule 144 may register a recognition result having a higher confidenceout of the current object recognition result and the previous objectrecognition result as the new current object recognition result.

That is, the object recognition module 144 may compare the tworecognition results, leave a recognition result with a higherconfidence, and compare the recognition result with the higherconfidence with a subsequent recognition result.

Meanwhile, the object recognition module 144 may determine a finalattribute of the object based on the plurality of recognition resultsfrom the sequential recognition (S2960).

While recognizing an object sequentially, the object recognition module144 may determine a final attribute of an object upon satisfaction of apredetermined condition. That is, the object recognition module 144 mayoutput, as a final recognition result, a current recognition result ofwhen the predetermined condition is satisfied.

After the predetermined condition is satisfied, an object recognitionprocess is no longer performed, and thus, a current recognition resultof when the predetermined condition is satisfied may be the last currentrecognition result.

For example, the last current recognition result at a time when objectrecognition is completed with respect to the entire object area, thatis, when recognition coverage of the object area reaches 100%, may bedetermined as a final attribute of the object.

In some embodiments, it may be set to finish the object recognitionprocess when the recognition coverages reaches a predetermined thresholdsuch as 90% and 95%.

Alternatively, the object recognition module 144 may compare the currentobject recognition result and a previous object recognition result, and,if the current object recognition result and the previous objectrecognition result are identical, the object recognition module 144 maymaintain the current object recognition result and change a confidenceinto an average between a confidence for the current object recognitionresult and a confidence for the previous object recognition result.

The present embodiment is an embodiment in which when the current objectrecognition result and the previous object recognition result areidentical, a confidence for the identical recognition results isdetermined as an average between the existing confidences of the tworecognition results.

Even in this embodiment, if the current object recognition result andthe previous object recognition result are not identical, the objectrecognition module 144 may perform control to register a recognitionresult with a higher confidence out of the current object recognitionresult and the previous object recognition as the new current objectrecognition result.

Meanwhile, the travel control module 141 may control driving of thetravel drive unit 160 based on a final attribute of the recognizedobject (S2980).

For example, if the recognized object is an object having a height thatthe mobile robot 100 is not able to climb, the travel control module 141may perform control to travel while detouring around the object.

In addition, if the recognized object is an object, such as a low bump,having a height that the mobile robot 100 is able to climb, such as alow door threshold, the travel control module 141 may perform control tokeep traveling in a straight forward direction.

In addition, if an object is recognized as an object that can restrictthe mobile robot 100 during traveling even though the object is a lowheight object such as a fan base, a human hair, a multi plug, anelectric wire, etc., the travel control module 141 may perform controlto travel while detouring around the object.

Meanwhile, depending on the determined final attribute of the object,the object recognition module 144 may register and manage thecorresponding object as a dangerous object or a non-dangerous object inthe map (S2970).

For example, if the recognized object is an object having a height thatthe mobile robot 100 is not able to climb or if the recognized object isan object that can restrict the mobile robot 100 during traveling eventhough the object is a low height object such as a fan base, a humanhair, a multi plug, an electric wire, etc., the object recognitionmodule 144 may register the recognized object as a dangerous object inthe map.

In addition, an object such as a bump that the mobile robot is able toclimb even when traveling in a straight forward direction may beregistered as a non-dangerous object in the map.

Thereafter, based on information on a dangerous object and anon-dangerous object registered in the map, the mobile robot 100 maytravel while detouring around the dangerous object.

FIGS. 30 to 34 are diagrams for explanation of a method for controllinga mobile robot according to an embodiment of the present invention.

Referring to FIG. 30, the mobile robot 100 may sense an object 3001while traveling. When the mobile robot 100 senses the object 3001, anobject area may be set and registered to include the surroundings of therecognized object.

Referring to FIG. 30, the object recognition module 144 may performcontrol to register, in a map to be stored in the storage 150, an objectarea 3010 having a predetermined size a×b around an object 3001 in anentire area 3000 is registered.

For example, the object area 3010 may be set to be in a rectangularshape of 1 m×1 m having each front/rear edge of 50 cm and eachleft/right edge of 50 cm around the object 3001.

Meanwhile, in addition to the registration of the object area 3010, theobject recognition module 144 may perform control to store positioninformation of the mobile robot 100 of when the object 3001 isrecognized. Meanwhile, the position information of the mobile robot 100may be information based on odometry information.

For example, the object recognition module 144 may perform control tostore position information of an object and position information of themobile robot in the following save format.

Save Format: (Op, Rp),

Op(X,Y)=Position of Object

Rp(X,Y)=Position of Robot

Meanwhile, the image acquisition unit 120 may acquire a plurality ofimages in the object area 3010 during traveling.

The image acquisition unit 120 may acquire a plurality of images bycontinuously photographing in the object area 3010 before the mobilerobot 100 departs out of the object area 3010.

FIG. 31 shows that a predetermined area of the object area 3010 isrecognized by two times of photographing. Circles 3110 and 3120 in FIG.31 shows recognition coverages, and sizes and shapes thereof may differdepending on configuration and performance of the image acquisition unit120.

Meanwhile, the mobile robot 100 may photograph the object 3001 at adifferent position in the object area 3010.

According to the present invention, by acquiring images of the object3001 at different positions during movement in the object area 3010 andperforming an object recognition process with respect to each acquiredimage, it is possible to enhance object recognition accuracy.

Meanwhile, the object recognition module 144 may manage the inner areaof the object area 3010 by dividing the inner area in a grid patternwith constant vertical and horizontal spacing.

FIG. 32 shows an example of an object area 3200 in 4×4 grid pattern.

As an image of the inside of the object area 3200 is photographed, theobject recognition module 144 may distinguishably store an area in whichan object is recognized.

For example, as shown in (a) of FIG. 32, it is possible to differentiatea recognized area and a non-recognized area in the entire object area3200 by changing a value of a predetermined area 3210 which has beenrecognized, and it is possible to determine the extent to which arecognition process has been proceeded in the entire object area 3200.

Meanwhile, depending on traveling and object recognition of the mobilerobot 100, the number of recognized areas 3210, 3220, 3230, and 3240 mayincrease, as shown in (b) of FIG. 32.

In addition, as shown in (c) of FIG. 32, when the recognition coveragereaches 100%, that is, when the object recognition process is completedin the entire object area 3200, the object recognition process may beterminated.

Alternatively, according to a setting, when the recognition coveragereaches 90% or 95%, the object recognition process may be terminated.

FIGS. 33 and 34 show part of a sequential object recognition processwith respect to the same object according to an embodiment of thepresent invention.

Referring to FIGS. 33 and 34, firstly, the object recognition module 144may recognize an image acquired at a first position Rp1 and recognize anobject as a fan. For example, a current recognition result indicative ofrecognition of a fan may have a confidence of 0.8.

Meanwhile, the aforementioned recognition result is the firstrecognition result and no other comparative subject exists, andtherefore, a registration result corresponding to the currentrecognition result may be a fan.

Meanwhile, secondly, the object recognition module 144 may obtain arecognition result indicative of a fan with a confidence of 0.7, basedon an image acquired at a second position Rp2 of the mobile robot.

Since the first recognition result and the second recognition result areidentical, the object recognition module 144 may maintain a fan as theregistration result corresponding to the current recognition result.

In this case, one of confidences for the two recognition results may beused as a confidence for the current recognition result.

Alternatively, as shown in FIG. 33, 0.7 which is an average ofconfidences of the first recognition result and the second recognitionresult may be used as a confidence for the current recognition result.

More preferably, since the first recognition result and the secondrecognition result identically indicate a fan, a confidence of 0.835which has been increased with a predetermined weight applied, as shownin FIG. 25, may be used as a confidence for the current recognitionresult.

Referring to FIGS. 33 and 34, based on an image acquired at a thirdposition Rp3 by the mobile robot, the object recognition module 144 mayobtain a recognition result indicative of a lamp base with a confidenceof 0.7.

Referring to FIGS. 33 and 34, the object recognition module 144 maycompare the third recognition result with a previous recognition result.

As the confidence of 0.7 for the third recognition result is lower thana confidence of 0.75 or 0.835 for a previous recognition result, theobject recognition module 144 may select a fan having a higherconfidence as a registration result corresponding to the currentrecognition result.

According to at least one of the embodiments of the present invention,since a mobile robot is capable of determining an attribute of an objectand adjusting a traveling pattern according to the attribute of theobject, it is possible to perform highly reliable object recognition andavoidance travel.

In addition, according to at least one of the embodiments of the presentinvention, it is possible to provide a mobile robot which is capable ofmoving forward, moving backward, stopping, detouring, and the likeaccording to a recognition result on an object to enhance userconvenience and improve operational efficiency, and cleaning efficiencyand a method for controlling the mobile robot.

In addition, according to at least one of the embodiments of the presentinvention, it is possible to provide a mobile robot and a method forcontrolling the mobile robot, the mobile robot which is capable ofaccurately recognizing an attribute of an object based on learningmachine.

In addition, according to at least one of the embodiments of the presentinvention, it is possible for a mobile robot to perform machine learningefficiently and extract data for recognition

As can be seen from the foregoing, the mobile robot according to thepresent invention is not limited to the configuration and method of theembodiments described above, but all or some of the embodiments may beconfigured to be selectively combined such that various modifications ofthe embodiments can be implemented.

Meanwhile, the mobile robot according to the embodiments of the presentinvention may be implemented in a recording medium, which may be read bya processor provided in a network device, as a code that can be read bythe processor. The recording medium that can be read by the processorincludes all kinds of recording media in which data that can be read bythe processor are stored. Examples of the recording medium include aROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical datamemory, etc., and may be implemented in a shape of carrier wave such astransmission through Internet. Also, the recording medium that can beread by the processor may be distributed in a computer system connectedthereto through the network, whereby codes that can be read by theprocessor may be stored and implemented in a distributive mode.

While the disclosed embodiment has been particularly shown and describedwith reference to exemplary aspects thereof, it will be understood thatvarious changes in form and details may be made therein withoutdeparting from the spirit and scope of the following claims, and Itshould be understood that many variations and modifications of the basicinventive concept described herein will still fall within the spirit andscope of the present invention as defined in the appended claims andtheir equivalents.

What is claimed is:
 1. A mobile robot comprising: a travel drive unitconfigured to move a main body; an image acquisition unit configured toacquire an image of surroundings of the main body; a storage configuredto store the image acquired by the image acquisition unit; a sensor unithaving one or more sensors configured to sense an object during themovement of the main body; and a controller configured to performcontrol to extract a part of the image acquired by the image acquisitionunit in correspondence to a direction in which the object sensed by thesensor unit is present.
 2. The mobile robot of claim 1, wherein thecontroller comprises: an object recognition module configured torecognize an object in the extracted part of the image based on datapre-learned through machine learning; and a travel control moduleconfigured to control driving of the travel drive unit based on anattribute of the recognized object.
 3. The mobile robot of claim 2,wherein the controller further comprises an image processing moduleconfigured to extract a part of the image acquired by the imageacquisition unit in correspondence to the direction in which the objectsensed by the sensor unit is present.
 4. The mobile robot of claim 2,further comprising a communication unit configured to transmit theextracted part of the image to a predetermined server, and receivemachine learning-related data from the predetermined server.
 5. Themobile robot of claim 4, wherein the object recognition module isupdated based on the machine learning-related data received from thepredetermined server.
 6. The mobile robot of claim 1, further comprisingan image processing unit configured to extract a part of the imageacquired by the image acquisition unit in correspondence to thedirection in which the object sensed by the sensor is present.
 7. Themobile robot of claim 1, wherein the controller is further configuredto: when the object is sensed from a forward-right direction of the mainbody, extract a right lower area of the selected image acquired at thespecific point in time; when the object is sensed from a forward-leftdirection of the main body, extract a left lower area of the selectedimage acquired at the specific point in time; and when the object issensed from a forward direction of the main body, extract a centrallower area of the selected image acquired at the specific point in time.8. The mobile robot of claim 1, wherein the sensor unit comprises: afirst sensor disposed on a front surface of the main body; and a secondsensor and a third sensor spaced apart in left and right directions fromthe first sensor.
 9. The mobile robot of claim 8, wherein the controlleris further configured to extract an extraction target area from theimage acquired by the image acquisition unit by shifting the extractiontarget area in proportion to a difference between a distance from thesensed object to the second sensor and a distance from the sensed objectand the third sensor.
 10. The mobile robot of claim 1, wherein aplurality of continuous images acquired by the image acquisition unit isstored in the storage, and wherein the controller is further configuredto extract, based on a moving direction and a moving speed of the mainbody, a part of an image acquired at a specific point in time earlierthan an object sensing time of the sensor unit from among the pluralityof continuous images.
 11. The mobile robot of claim 10, wherein thecontroller is further configured to extract a part of an image acquiredat a point in time earlier than the object sensing time of the sensor asthe moving speed is slower.
 12. The mobile robot of claim 1, wherein thecontroller is further configured to store position information of thesensed object and position information of the mobile robot in thestorage, and perform control to register an area having a predeterminedsize around a position of the sensed object as an object area in a map.13. The mobile robot of claim 12, wherein the controller is furtherconfigured to recognize an attribute of an object sequentially withrespect to images acquired in a predetermined object area by the imageacquisition unit, and determine a final attribute of the object based ona plurality of recognition results from the sequential recognition. 14.A method for controlling a mobile robot comprising: sensing, by a sensorunit, object during movement; extracting a part of an image acquired byan image acquisition unit in correspondence to a direction in which theobject sensed by the sensor unit is present; recognizing an object inthe extracted part of the image; and controlling driving of a traveldrive unit based on an attribute of the recognized object.
 15. Themethod of claim 14, wherein the recognizing of the object comprisesrecognizing an attribute of the object in the extracted part of theimage based on data that is pre-learned through machine learning. 16.The method of claim 14, wherein the extracting of the part of the imageacquired by the image acquisition unit comprises: when the object issensed from a forward-right direction of the main body, extracting aright lower area of the selected image acquired at the specific point intime; when the object is sensed from a forward-left direction of themain body, extracting a left lower area of the selected image acquiredat the specific point in time; and when the object is sensed from aforward direction of the main body, extracting a central lower area ofthe selected image acquired at the specific point in time.
 17. Themethod of claim 14, further comprising storing the extracted part of theimage.
 18. The method of claim 14, wherein the controlling of thedriving the travel drive unit comprises performing control to perform anavoidance operation when the sensed object is not an object that themobile robot is able to climb.
 19. The method of claim 14, wherein theextracting of the part of the image comprises extracting, based on amoving direction and a moving speed of the main body, a partial area ofan image acquired at a specific point in time earlier than an objectsensing time of the sensor unit from among the plurality of continuousimages.
 20. The method of claim 14, further comprising: when the sensorunit senses an object, storing position information of the sensed objectand position information of the mobile robot in the storage andregistering an area having a predetermined size around a position of thesensed object as an object area in a map; recognizing an attribute ofthe object sequentially with respect to images acquired by the imageacquisition unit during movement in the object area; and determining afinal attribute of the object based on a plurality of recognitionresults from the sequential recognition.